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Introduction 

Using evaluation to improve performance has become a priority for OECD 
policy makers and practitioners in the field of public information/communication 
and education about development. They want to increase the effectiveness of public 
awareness and learning activities about development because much-needed political 
will for an ambitious agenda for reform requires informed public support. However, 
if citizens in OECD countries are increasingly conscious of global interdependence 
they do not understand much about the causes and implications. Furthermore, 
official donors are increasing resources to inform and educate citizens about global 
development in a policy environment that is focused on demonstrating results. 
The learning obtained, and results identified, through evaluation puts it at the 
heart of this agenda. There is, however, some inertia in turning the principle of 
evaluation into practice 

How can this consensus about the importance of evaluation be transformed 
into action? It seems that the key actors in this sector — both government and 
non-governmental — must first address a variety of evaluation challenges ranging 
from the asymmetric power relationship between the institutions that fund, and 
implement, these activities and evaluations; to the diverse nature and understanding 
of communication and education about development; to limited financial resources 
for evaluations. 

Achieving a shared understanding about evaluation was the aim of the 
Informal OECD Experts’ Workshop on “Development Communication, Advocacy 
and Education:Tips and Tools for Improving Evaluation”, which was hosted by the 
German Ministry for Economic Co-operation and Development (BMZ), and took 
place in Bonn on 19 and 20 March 2007. Evaluation experts from the education, 
communication, advocacy and development co-operation communities came 
together with policy makers and practitioners who deal with public awareness 
raising and learning about developmentThe workshop provided a space to learn 
about different views and expectations from evaluation and the lessons learnt on 
how to overcome some of the challenges. 

This Policy Brief is an introduction to evaluation, with proposals on how 
official donors and civil society organisations can overcome some key evaluation 
challenges. The concrete experiences and lessons learnt at the Bonn workshop are 
presented in the context of why it matters for OECD countries to improve their 
performance in informing and educating citizens about global development issues. 
Three approaches to awareness raising and learning — information/communication, 
advocacy and education — are included, because, as a first step, it is useful to 
demonstrate the interlinkages and complementarities between them. The Policy 
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Brief should serve as a useful reference and guide for official donors using evaluation 
to leverage institutional support and increased financial resources for this work. It 
should also enable greater collaboration between donors and practitioners whose 
work is financed by official development assistance (ODA) thanks to improved 
understanding of their respective perspectives and evaluation objectives. This 
Policy Brief is an introduction to evaluation. It does not delve into the complexities 
and specificities of what constitutes best practice in information/communication, 
advocacy and development education. Nor does it provide guidelines for evaluation. 
This could be addressed in future work. 



Public Awareness Raising: Definitions 

A diversity of activities is undertaken in the name of public awareness raising 
and learning about development in OECD countries. While there is a general 
understanding of what constitutes an informational or educational activity, evaluation 
requires precise definitions.The three main categories of awareness-raising activities 
about development and global issues in donor countries are: i) development 
information/communication; ii) advocacy and campaigning; and Hi) development 
and global education.This section outlines their specificities and similarities because 
they should be factored in to decisions about evaluation. 

Development Information! Communication 

This is communication about aid and development challenges, policies, 
programmes, and results to different audiences in donor countries. Official donors and 
NGOs implement information/communication strategies to exist, to be transparent 
and accountable, and to raise aid funds. Also called public affairs such strategies 
have existed since the early years of ‘institutionalised’ development co-operation 
policy in the 1960s'. In addition to raising “corporate awareness” these strategies 
often try to address misperceptions about realities in developing countries and 
to build and sustain public and political support for donors’ development policies 
and aid budgets. 

Transparency and accountability obligations in OECD democracies underpin 
development information/communication. As the right of access to information 
— most notably through freedom of information acts (FOIs) — has increased in 
OECD countries, so has the need for more active and transparent communication 
with taxpayers. Diminished public trust in public policy making is an added incentive 
for governments to open up and engage with citizens. Greater understanding about 
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development policies and associated challenges help to raise public support and 
confront institutional resistance to tough reforms^. 

National and international NGOs also contribute to increased public pressure 
on governments to be more transparent about aid policy, flows and results in 
developing countries^ Governments are thus required to respond and pre-empt 
demand for this information and present the results achieved with ODA. 

Over the past decade, the nature of public communication in development co- 
operation has evolved from being unidirectional in the form of glossy annual reports 
and press releases to a more interactive approach using technologically sophisticated 
and participatory processes. Internet-based and face-to-face consultations between 
development professionals and members are increasingly popular. Nevertheless, 
progress remains uneven across OECD countries. 

_ , , Box I . Advocacy for Development 

Advocacy and 

Campaigning 

Advocacy (the act 
of arguing on behalf of a 
particular issue, idea or 
person) and campaigning 
(an organised effort 
towards specific political 
goals) occupy more space 
in international affairs than 
ever before. Advocacy and 
campaigning have grown 
in frequency and numbers 
since the late 1980s when 
international development 
NGOs and other social 
movements stoked public 
debate about World Bank lending policies (most notably against structural 
adjustment policies). This was followed by large mobilisations against the World 
Trade Organisation in the late 1990s (Seattle, Turin, etc.), the Jubilee Campaign for 
debt relief leading up to 2000, and G8 Summits in the new millennium"^. 

Indeed, advocacy has become a core activity for NGOs — North and South — 
as evidenced by the Global Call to Action against Poverty and other international 
networks such as the Reality of Aid. Moreover, advocacy is no longer the brain- 
child of non-governmental actors as demonstrated by the historic creation of the 
United Nations Millennium Campaign in 2001. This campaign is mandated by the 

8 



Tibbett (2007) defines advocacy as the following: 

— Advocacy is about change. It is strategic. It is planned. 
It should have clear objectives. 

— Often in contemporary advocacy, organisations and 
campaigners facilitate people to speak for themselves: 
‘agency of the poor’. 

— Campaigning is a subset of advocacy. 

— Advocacy is an emerging discipline in development. It is 
still finding its feet. It is more an art than a science. 

— While communications may be about the status 
quo or preserving something. Advocacy is about 
change. 

— Communications can support advocacy objectives. 

Source: see SteveTibbett’s presentation at the Informal Experts’ Workshop on 
“Development Communication, Advocacy and Education: Tips and Tools 
for Improving Evaluation” vwvw.oecd.org/dataoecd/43/3 l/38406274.pps 
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United Nations to advocate and campaign for the achievement of the MDGs^ 
National governments also play an important catalytic role for advocacy about 
development issues: it was Prime Minister Tony Blair’s invitation to British civil 
society organisations (CSOs) to build a campaign for the United Kingdom’s G8 
Presidency in 2005 that led to the Make Poverty History Campaign^. In many DAC 
countries ODA finances civil society campaigns on development issues. Trends 
suggest that both government and NGO campaigns for development will continue 
to take more space in contemporary development co-operation. 

Development/Global Education 

Development and global education prepares humans to respond to the 
challenges of an interconnected world, to take responsibility for, and to advocate, 
global solidarity and social justice (Bourn, 2007; Asbrand/Scheunpflug, 2006). 
Citizens, it is argued, have the “right” to development/global education because: 

1) It equips them with tools for understanding an interdependent world with 
“global public bads” such as poverty, global warming, wars and conflicts. 
Through greater understanding they may promote public goods such as 
wealth, peace and participation (UNDP, 1999). 

2) It enables them to come to terms with the uncertainty brought about by the 
acceleration of social change and by the fragmentation and the hybridisation 
of cultures and lifestyles (Appadurai,l990). 

3 ) It helps them to empathise with strangers and to see the world from 
the perspectives of, for example, minorities, rural populations, and the 
underprivileged and economically excluded. 

4 ) Governments committed to it as part of the UN Decade of Education for 
Sustainable Development^. 

Box 2. Definition of Development Education by European NGOs 

Development education is an active learning process, founded on values of solidarity, 
equality, inclusion and co-operation. It enables people to move from basic awareness 
of international development priorities and sustainable human development, through 
understanding of the causes and effects of global issues, to personal involvement and 
informed action. 

Development education fosters the full participation of all citizens in world-wide 
poverty eradication, and the fight against exclusion. It seeks to influence more just 
and sustainable economic, social, environmental, and human rights based national and 
international policies. 

Source: European Development Education Forum, vwvw.deeep.org/english/whatJs_de/definitions/. 
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OECD governments promote development/global education by providing 
a policy framework and financial support for it through their ministries/agencies 
for development co-operation. In most countries, donors outsource development 
education to NGOs and other CSOs. Indeed, development education policies and 
budgets often exist because of lobbying and demand from civil society. At present. 
Ministries of Education are glaringly absent from this scene although efforts are 
under way in some countries, including Finland, Ireland, Sweden and the United 
Kingdom, to institutionalise teaching about global issues in the school curriculum®. 
Nevertheless, a high proportion of development education takes place in schools 
thanks to the personal motivation of teachers interested in global issues. 



Complementary but Different Approaches 



In practice, the three approaches — information/communication, advocacy 
and development education — are interlinked and overlapping: they all influence 
society towards greater social justice and they all deal with learning. Still, the 
theories and methods underpinning the approaches differ. Occasionally, this 
contributes to disagreement and tension between actors as they tend to promote 
and defend their preferred approach. 

For example, the fundamental philosophy of education, including development/ 
global education, is that free individuals must decide for themselves. Public education 
should offer controversy and multiple perspectives in a neutral, non-ideological 
manner to enable the individual make his/her own judgement. Yet, campaigning 
and communication are not bound by this philosophy: campaigning aims to arouse 
people’s enthusiasm for a specific perspective and policy change; development 
communication aims to inform and raise awareness. 



Table I. Information/Communication, Advocacy and Development/Global Education: 

Where They Differ 





Information/Communication 


Advocacy 


Development/Global Education 


Rationale 


Fill knowledge gaps 


Policy change 


Change individuals; answer 
learning needs in an 
interdependent world 


Objective 


• Transparency 

• Accountability 

• Gain support for reform 

• Corporate 
communication/public 
affairs: image, reputation, 
credibility 

• Contribute to behaviour 
change 


• Advocacy and campaign- 
ing to change policies 

• Mobilise citizens to sup- 
port change and social 
justice 


• Enable people to live 
in an interdependent 
world society and 
to understand social 
justice 


Key focus for 
the public 


To inform 


To engage in campaigning 
and action 


To learn 



10 



Building Public Awareness of Development: Communicators, Educators and Evaluation 



Who funds development education often drives the objectives of the activities 
undertaken and thus determines “indicators” of success. Typical indicators are 
increased public support for aid or debt cancellation. But this can cause conflict 
with educationalists who, for example, consider that successful learning does not 
necessarily result in more public support or public mobilisation. Success may be a 
learner who becomes more critical of the aid budget. Similar conflicts also occur in 
evaluation, which is regularly a donor condition and contributes to the perception 
that it is used for control rather than lesson learning or a combination of both. 



Evaluation of Public Awareness Raising and Learning 
about Development, Why Now? 

Two major forces are behind the unprecedented gains in prominence of 
communication, and the education of OECD citizens, about global development 
since 2000. Firstly, the adoption of the Millennium Development Goals (MDGs) and, 
in 2005, the Paris Declaration on Aid Effectiveness have established results oriented 
deadlines that require a major change in donor behaviour with simultaneous pressure 
to achieve development results for poor people in a relatively short period of time. 
Political will for this reform agenda needs to be sustained and OECD leaders held 
accountable to their commitments. More mobilised and supportive citizens are key 
determinants. Moreover, OECD governments must ensure that citizens have access 
to education that is relevant to the interdependent world they live in. Evaluation has 
come to the fore as communicators and educators need to learn from experience 
and show their added value in the face of more demand for public information and 
education about global issues. 

The MDGs, the Paris Declaration and Political Will 

An engaged and supportive citizenry could be the impetus for the successful 
reform of development co-operation in OECD countries. OECD governments 
need political will and support from decision makers to translate the ambitious 
commitments they endorsed at the UN Millennium Summit (MDGs) and the Paris 
High Level Forum on Aid Effectiveness into concrete policy changes and increased 
resources for development. Politicians, who are accountable to the voting public, 
can win public support for certain policies and reforms, but their actions are also 
responsive to public opinion. Caution is necessary when “following” public opinion 
because it is badly informed about aid and development co-operation. In order to 
win public support and to follow informed public opinion about aid and development 
both government and non-governmental actors need to better educate citizens 
about the reform agenda. 
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Effective development co-operation requires a complex mix of policy 
instruments where official development assistance (ODA) is but one pillar. However, 
opinion surveys from a variety of DAC countries find that only a minority of 
citizens make links between a donor country’s non-aid policies and development. 
Furthermore, promoting more policy coherence for development in national and 
international decision-making circles is politically sensitive as it addresses issues 
that have competing vested interests. The complexity of this agenda also makes 
the content of public campaigns and education more complex. 

When endorsing the Paris Declaration on Aid Effectiveness signatories 
reaffirmed their commitment to accelerate progress in implementing aid effectiveness 
commitments, “especially in enhancing donors’ and partner countries’ respective 
accountability to their citizens and parliaments for their development policies, 
strategies and performance’’^. This commitment provides an additional mandate 
to donors to communicate and educate OECD taxpayers about development. It 
also puts the spotlight on the quality, performance and results of all aid activities 
— including development communication and education. 

There is evidence that governments are responding to increased pressure 
to engage OECD citizens in development. By 2007, the UN Millennium Campaign 
was supporting MDG campaigns in partnership with official donors and CSOs in 
17 member states of the European Union (EU), Australia, Canada, Japan, New 
Zealand and the United States'^, with a positive impact on public awareness as 
Figure I shows. 



Sweden 

Netherlands 

Slovenia 

Denmark 

Belgium 

Finland 

Slovakia 

Austria 

Hungary 

Portugal 



Figu re I . Awareness of the MDGs 
Top Ten EU Member States (2004 & 2007) 




Source: OECD Development Centre based on Special Eurobarometer 280 (2007) and 222 (2005), European 
Commission, Brussels. 
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Between 2004 and 2006, the amount of ODA allocated to communication and 
education increased by approximately €60 million". Furthermore, there has been 
wide media coverage of image-conscious celebrities who alone, and in partnership 
with international campaigns by NGOs, are calling for policy change by donors 
and/or for more aid for Africa.This puts the spotlight on political leaders and their 
administrations and challenges them to be more effective in their own communications. 
The prioritisation of evaluation is also evident where certain large public campaigns 
were evaluated, most notably the Swedish MDG Campaign and the Make Poverty 
History Campaign. Still, evaluation has not become mainstream practice. 

Global Interdependence and Citizenship 

All world citizens are affected — sometimes positively — by globalisation 
and interdependence which are driven principally by economics and technology'^. 
Public debate in OECD countries shows that citizens are concerned about shifts 
in the global balance of power as countries in transition such as China and India 
gain in economic (and political) prominence, and about challenges to global human 
security and well-being from international terrorism, climate change, marginalisation, 
discrimination and corruption, and not least, from the extreme poverty in which 
20 per cent of the world’s population lives'^ 

OECD governments recognise the need to help citizens understand the 
world they live in and the challenges that come with it. Indeed, education for global 
citizenship is being institutionalised for this reason within a number of formal 
education initiatives, most notably in the United Kingdom as shown by recent 
revisions of the school curriculum. As the stakes are raised for public education 
about global and development issues, there will also be more pressure for greater 
professionalisation of development/global education among NGO practitioners. 
Evaluation helps identify best practice and contributes to more rigorous monitoring 
of results, capacity and lesson learning. 

Now that the context for increased public awareness and learning for 
development and the incentives for using evaluation to strengthen performance 
have been established, we can focus on the bottlenecks that obstruct more evaluation 
of public information/communication, advocacy and development education. 



Putting Evaluation Principles into Practice 

The introduction to our Policy Brief stated that transforming the consensus 
about the priority and need for evaluation into practice is not straightforward due 
to a number of specific challenges that policy makers and practitioners are having a 
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hard time resolving. Many of them are not new as demonstrated by earlier reports 
on these issues'"^. In this section the easier elements of putting evaluation into 
practice will be addressed before delving into some specific — tough — challenges 
raised at the workshop. 

Defining Evaluation 

The DAC Evaluation Network defines evaluation as the systematic and 
objective assessment of an ongoing or completed project, programme or policy, its 
design, implementation and results ‘^.The aim is to determine relevance, fulfilment of 
objectives, efficiency, effectiveness, impact and sustainability. An evaluation should 
provide information that is credible and useful, enabling the incorporation of lessons 
learned into the decision-making process of the practitioner and the donor. 

In the field of information/communication, advocacy and education, evaluation 
is a tool that serves a number of purposes including accountability to donors 
and target groups as well as the people who are represented in these activities 

— usually poor people from developing countries. While warranted, accountability 
to funders often overshadows other valid purposes of evaluation such as learning 
and improving practice. For practitioners, therefore, an evaluation should: 

— be systematic; 

— be an evidence-based reflection on practice; 

— formulate clear consequences for practice in the future; 

— enable continuous learning, improvement and evolution in the practitioners’ 
own work'^. 

In development education, for example. Bourn (2007) states that: “evaluating 
and measuring success of the impact of global and development education 
programmes can only be located within the learning processes and learning. We 
can and should not lose sight of the relationship between the ‘development’ agenda 
and the ‘learning agenda’”'^ 

Moreover, advocacy and development education have evolved from practice 
rather than theory. A feature of this type of evolution is the absence of robust 
research and evidence on best practice. Evaluation can contribute to filling this 
gap but it must not lose sight of immediate needs such as identifying results and 
lessons learnt for practitioners. 



14 



Building Public Awareness of Development: Communicators, Educators and Evaluation 



Getting the Basics Right 

The basic rules of evaluation are relatively straightforward and should not be 
a major stumbling block to evaluation newcomers.The most rudimentary point is 
that an evaluation, whether external or internal (self), should be systematic. For this 
reason it may be helpful to follow the different steps or stages of an evaluation as 
shown in the evaluation cycle (Figure 2). It may also be useful to refer to material 
and guidelines produced by evaluation associations and networks, which exist in 
most OECD countries and internationally. Box 3 gives an overview of evaluation 
associations that produce such material and guidelines. 



Figure 2. The Evaluation Cycle 




Sources: Adapted from Rolff (1998); Stufflebeam/Shinkfield (2007). 
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Box 3. Standards for Evaluation 

International standards outline transparent rules and expectations from evaluations. It 
may be helpful to use these for evaluations of development communication, advocacy and 
education, even if they had not been conceptualised for this purpose. National ministries 
and national evaluation associations also provide helpful standards on evaluation. 

DAC Standards: DAC Evaluation Quality Standards 

vwvw.oecd.org/document/29/0,2340,en_2649_34435_ 1 6557 1 49_ I _ I _ I _ 1 ,00.html 
These standards are in a test phase of application.The DAC Evaluation Quality Standards 
identify the key pillars needed for a quality evaluation process and product. They 
are available in both English and French. These standards focus on the effectiveness 
of international development programmes and may provide useful perspectives for 
development communication, advocacy and education. They describe the rationale, 
purpose and objectives of an evaluation, evaluation scope, context, methodology, 
information sources, independence, evaluation ethics, quality assurance, relevance of 
the evaluation results, and completeness. 

Joint Committee on Standards for Educational Evaluation 

vwvw.vvmich.edu/evalctr/jc/ 

The Joint Committee has set out standards on evaluation, especially for educational 
purposes.They differentiate between utility, feasibility, propriety and accuracy standards.The 
publication The Program Evaluation Standards is very helpful because it gives examples for 
every standard and a helpful reference. It is especially useful for beginners in this area. 
See: James R. Sanders, Chair (ed.) (2003): The Program Evaluation Standards, 2nd edition. 
Sage Publication, Thousand Oaks, USA. 

Standards for Evaluation in the UN System (2005) 

http://vwvw.uneval.org/index.cfm?module=Library&page=Document&DocumentlD=5662 
Very concise, the UNEG Standards focus on the institutional framework and management 
of the evaluation function, on competencies and ethics, on conducting evaluations and 
on evaluation reports. 

This brochure is available in English, Spanish, French, Russian, Chinese and Arabic. 

Independent Evaluation Group, World Bank Group 

vwvw.worldbank.org/ieg/ecd/tools/ 

Monitoring and Evaluation (M&E): Some Tools, Methods and Approaches 
An overview of a sample of M&E tools, methods, and approaches including their 
purpose and use; advantages and disadvantages; costs, skills, and time required; and 
key references. 

Available in English, French, Portuguese, Russian, Spanish and Arabic. 



Lightweight Challenges 

The more lightweight challenges relate to the evaluation process. In general, 
they can be overcome relatively easily.They are:/) getting started and consultation 
with key stakeholders to determine the evaluation question;//) identifying criteria 
and indicators;///) choosing the methodology; and iv) reporting and communicating 
the results of the evaluation. 
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i) Immense effort is often required to start an evaluation. It is tempting to 
want to evaluate everything but experts warn against this because it is often 
too ambitious and impossible to achieve'®. In a resource-scarce sector, the 
evaluation must be relevant to daily workThe context of the whole work can 
be considered in the evaluation but it should nevertheless focus on a specific 
question. Once the decision has been made to evaluate an activity, there should 
be consultation with key stakeholders who are responsible for the activity 
and transparency throughout the process.Together, decisions should be taken 
on what will be evaluated and how to articulate the evaluation question(s). 

ii) Criteria and indicators should then be identified and the evaluation should 
stick to measuring them. Criteria (the characteristics/answers to the evaluation 
question being asked) and indicators (measures that demonstrate whether 
the criteria have been met or not) of the evaluation are the framework 
for the whole evaluation. Indicators are often figures, but this is not always 
necessary. For example, the criteria for an evaluation investigating the quality 
of messages on poverty could be that the multidimensional nature of poverty 
is communicated. Indicators could be the number of times different dimensions 
of poverty are referred to in messages, articles and speeches'^. 

Hi) When choosing the methodology that will enable the evaluator to investigate 
the evaluation question and collect data, look for efficient and appropriate 
methods of data collection and remember that even if the data seems limited 
it may be sufficient to answer the evaluation question. Evaluation professionals 
regularly comment that it is not necessary to collect new data. There is a 
tendency to forget about data at hand such as emails, internal communications, 
newsletters and registrations for meetings.Therefore, it is not always necessary 
to commission expensive representative questionnaires. It is useful to integrate 
data collection for the evaluation into daily, long-term work so that when 
evaluation is underway the extra work generated is diminished. Once the 
data are collected the findings should be interpreted with the help of the 
evaluation criteria. It is important to restrain subjective judgements about 
the quality of the work. Basing judgements on criteria and indicators will be 
more objective. 

iv) Reporting, communication and discussion of the results constitute the most 
neglected part of the evaluation process. Yet it is probably the most crucial 
stage. Discussion of the results and conclusions of the evaluation — with all 
relevant actors — is the occasion to identify lessons that can be applied to 
future activities. In practice, open discussions with all actors about evaluation 
results can be tense, especially where there is weak mutual trust. Tensions 
and fears about negative results are commonplace when future funding 
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depends on the success of the activity. This can be alleviated if the evaluator, 
the practitioner and the donor agree on the purpose of the evaluation, the 
criteria and indicators, from the very start. 

From Lightweight to Heavyweight Challenges 

Tackling the heavyweight challenges is a more difficult task because they 
are about expectations, power and relationships, trust and legitimacy. Typical 
tensions underlying evaluations, as identified by the Belgian training centre for 
development, ITECO, include^°: 

— Is it objective or subjective? 

— Should it be external and/or internal? 

— Is it about results and/or process? 

— Is it for control and/or learning? 

— Can it contribute to individual and/or collective change? 

Furthermore, official donors are preoccupied with two types of challenges: 
i) using evaluation to measure attribution (causality between the action and an 
observed change in attitude or behaviour) and long-term effects; and ii) how to 
develop the most coherent and appropriate evaluation policy for short-term 
awareness raising and learning activities outsourced to NGOs and grass-roots 
organisations. 

Finally, participants to the Bonn workshop tabled questions about evaluation 
such as: 

— Can one evaluation serve accountability and lesson learning? 

— What is a good evaluation? Can it answer all questions? 

— Who should finance the evaluation? Who should do it? When should it be 
done? 

As there is some overlap between these challenges we present five categories 
of challenges and make suggestions on how to deal with them. 

Challenge I: Can results identified by the evaluation be attributed to 
specific activities? 

The bottom line is that evaluation will not always solve the attribution 
challenge. In all three approaches to public awareness raising and learning, it is 
particularly difficult to determine a clear causality between concrete activities 
and the expected outcome in public opinion, policy change and learning. The 
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drivers of attitudinal and behaviour change can be opaque. But we heard from one 
participant to the workshop that while attribution is definitely difficult to determine 
from evaluation, this is not an acceptable excuse for senior policy makers. It was 
suggested that one way of determining causality is to conduct baseline research 
on attitudes and behaviour patterns of target groups before the project starts. 

In some cases scientific research is better able to measure the causality 
between learning and attitudinal change. While the learning environment may 
have an identifiable influence on the learner we also know that people may learn 
very quickly, easily or slowly, and this depends on the quality of education, their 
learning capacities, their peers as well as public debate (see Box 4). 

The disadvantages of using baseline and scientific research to solve the 
attribution challenge are cost and the time-intensive nature of scientific 
research. 



Box 4. The Complexity of Learning 

Contemporary cognitive psychology and neuro-science conclude that learning is a 
complex self-led and self-constructive process which is caused not only by the input but 
by internal self-related structures for dealing with knowledge and impressions (National 
Research Council, 2000; OECD, 2002). In some situations very little detail may have a 
tremendous effect, leading to a self-related learning cascade; in other cases there is no 
resonance with the learner.The non-formal learning situation and the complex learning 
tasks that are set by development communicators, campaigners and educators, such as 
developing critical thinking about development, makes clear attributions between their 
efforts and the learning outcome particularly challenging. 



Challenge 2: What should an evaluation measure? 

Evaluations should measure process and results. Unlike attribution, which 
should be handed over to researchers, evaluation is effective in measuring i) the 
resonance of an activity with recipients; ii) the quality of concepts and processes; 
and Hi) the efficiency of the resources spent. 

— Resonance can be measured by data from an activity such as the number of 
people reached, public opinion about the topic addressed, newspaper articles, 
feedback from participants in an event, perspectives from stakeholders 
etc. 
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— The quality of the process may be assessed by evaluating concepts used in 
comparison to the national or international discourse about the concepts 
and evaluating how the activity was implemented, how people were involved, 
the methods and strategy used. 

— An efficiency analysis focuses on the resources spent in relationship to the 
nature of the activity and resources used for similar activities. 

The same evaluation questions can be shared across the three public awareness 
raising approaches for process evaluations and when evaluating the results achieved. 
The question should be different when evaluating an activity’s overall objective 
and the longer-term outcomes. Core objectives of a communication strategy, 
such as improved image, will be evaluated differently to the advocacy objective of 
policy change and the development education objective of motivating people to 
learn about development. 

Challenge 3: What are the conditions for a good evaluation? 

Standard benchmarks and money are two conditions for a good evaluation. 
Both are scarce in the public awareness and learning sector. 

Weak theoretical concepts, and the lack of an empirical base in the form 
of agreed-upon good practice, make it difficult to define criteria on quality. A 
practical way of overcoming the lack of standardised quality guidelines when setting 
criteria and indicators is to take obvious structural elements of these activities 
into account. For example, quality criteria for activities done by a voluntary 
network will be different to those used to evaluate activities by trained and paid 
professionals. The same applies to the environment (e.g. formal or non-formal 
education) in which the work is done and the type of target groups (e.g. school 
children or adult learners). 

There is inadequate funding for evaluation and thus less investment in all stages 
of the evaluation cycle. This is due to the lack of, and the nature of, funding for public 
awareness work in general. A frequent question in discussions about evaluation 
is how much should be reserved for evaluation. Should it be, for example, a fixed 
percentage of the project cost? There is no established rule for this. However, 
DFID (2005) provides useful orientation: “Opinions vary as to what percentage 
of your budget should be dedicated to evaluation. A rough rule of thumb would 
be between 5-10 per cent. For small projects and those with short time-frames, 
evaluation might take up a larger part of the budget. This might also be the case 
with pilot projects, which try to determine how successful a programme will be 
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if it is rolled out at a later date. In cases like this, evaluation costs might take up 
30 per cent and more of the total budget.” 

The nature of funding — government is the major donor — is a particular 
concern of campaigners and development educators because there is often a 
mismatch between the donor’s objective for an evaluation and the practitioner's^'. 
The evaluation is often perceived by practitioners as a tool to cut funding rather 
than an incentive to learn from evaluation to increase quality. Consequently, 
practitioners have few incentives to develop a culture of evaluation in their work. 
To overcome this challenge, donors should request and finance evaluations that 
meet the practitioners’ learning needs as well as the donor’s accountability needs. 
Practitioners, on the other hand, must be willing to learn from mistakes. 



Challenge 4: When is self-evaluation acceptable? 

Self-evaluation is organised by those responsible for and implementing the 
project. Conscious of the tension between objectivity and subjectivity in evaluation, 
cash-strapped communicators and educators ask when, and what type of, self- 
evaluation is acceptable and justifiable. The crude answer to this question is that 
it depends on the organisation, the activity and the purpose of the evaluation. It 
is also important to consider the advantages and disadvantages of self-evaluation 
versus external evaluations. 

The decision between self-evaluation and external evaluation should betaken 
by balancing the advantages and disadvantages. Make the most of self-evaluation 
by being systematic and being explicitly bound to an evaluation cycle. This means 
treating the identification of criteria, indicators, and reporting and planning as 
carefully as in external evaluations. For first-time self-evaluators it may be helpful 
and worthwhile to seek advice or supervision by an expert evaluator. 
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Table 2. The Advantages and Disadvantages of Self-evaluation 
Versus External Evaluations 



Type of Evaluation Advantages Disadvantages 



• The objectives of the evalua- 


• Self-evaluators risk being 


tion and the evaluation 


biased and subjective. 


process are decided by and 
usually well-known by the 
core actors and it should 
fit well with their work 
mode. 


• In understaffed organisa- 
tions, it negatively affects 
capacity to implement the 
project. 


• Self-evaluation enables 
deeper identification 

Self-evaluation with the work and easier 


• Training to enable greater 
objectivity and distance 
from the work may be 
required. 


implementation of the 


recommendations. 


• If personnel are new to 


• A good self-evaluation gives 
new impetus to a project. 


self-evaluation, it may cost 
as much as an external 
evaluation. 




• Decisions are often driven 




by the cost factor but this is 
not the most advantageous 
aspect of self-evaluation. 



• An independent expert 


• The evaluator might not 


evaluator conducts it and 


be an expert in awareness- 


is more likely to have a 


raising approaches and 


distant view and provide 


may not be familiar with 


objective feed-back. 


evaluation methods for 


• Outside perspectives 


that particular approach. 


may help to find new 


• Criticism - even by a critical 


External evaluation perspectives and ideas for 

the work. 


friend - may be stressful. 

• It can take a lot of time 


• Experienced evaluators may 


and energy to provide 


find it easier to focus on the 


information to the 


most important questions 


evaluators. 


and stay independent. 


• Follow-up may not be easy 




once the evaluator has 
moved on. 
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Challenge 5: Is evaluation always the right tool? 

Expectations from evaluation tend to be too high. Better use of evaluation can 
improve the quality, and demonstrate the immediate results, of public awareness- 
raising work. However, demands from evaluation often exceed its capacity, especially 
in terms of attributing the impact of awareness-raising strategies to specific activities, 
and more long-term changes in attitudes and behaviour. Expectations from evaluation 
must, therefore, be better managed. 

Public awareness policy makers and practitioners must effectively manage 
expectations about impact within their own organisations because it is difficult 
and costly to decipher impact through evaluation. An approach to take is to stress 
the potential, and the limits, of evaluation in this sector. Clearly, evaluation improves 
the quality of activities by investigating the validity of concepts, the efficiency of 
processes and the immediate results. By effectively applying the results and lessons 
from such evaluations the gains in quality will pave the way for achieving the long 
term “impact” of public awareness-raising activities. 

Good Practice: Evaluation of In formationICommunication, Advocacy 

and Development Education 

This section presents examples of good practice in evaluation for the three 
approaches to public awareness and learning. One evaluation tool that addresses 
the issue of how to evaluate the results of a comprehensive strategy, over the long 
term, is also provided. 

Evaluation of communication: the M/7/enn/um Development Goals 

The adoption of the MDGs in 2000 provided official aid agencies, NGOs and 
other development actors with a new framework for building public awareness 
about development challenges. It represented no less than a tectonic shift in the 
way donors communicate about development: away from inputs to messages 
about “real” results. Many official aid agencies seized the promise made by world 
leaders to halve world poverty by 2015 to communicate in a different way to 
taxpayers. Through greater awareness about the goals, citizens may start holding 
their governments to account for their commitments to poor people. 
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Good Practice I: Swedish MDG Campaign “The Chance of a Life-time” 

Sweden was one of the first DAC members to build and implement an MDG 
communication strategy. With a long-term key message:“lt’s possible to halve poverty 
by 2015”, different MDG themes were communicated each year from 2003 to 2005. 
An evaluation was commissioned in 2005 with two objectives: 

1) to provide input to the Swedish Ministry of Foreign Affairs on how to proceed 
with future communication about the MDGs; 

2) to evaluate how the project group had succeeded in their mission.Were the objec- 
tives of the campaign reached? 

The tools used were both qualitative and quantitative: in-depth interviews with 
people involved in the campaign and partners, and telephone questionnaires with 
289 politicians and opinion leaders. The second part of the evaluation was desk 
research looking at goals, activities and outcomes, weighing them up with the objective 
on how the project group would work together. 

The most challenging part of the evaluation was the latter part due to difficulties in 
grasping all the work and activities of the project group over three years. 

The evaluation was communicated to and discussed with all partners involved as well 
as the Minister and State Secretary. It was also translated into English and shared with 
the UN Millennium Campaign. 

While the objectives of the campaign were ambitious it was possible to evaluate the 
campaign because its objectives were clear, and there was learning and follow-up 
throughout the campaign period itself. 

Source: Presentation by Ann Lundqvist at the Experts’ Workshop, “Development Communication, Advocacy 
and Education: Tips and Tools for Improving Evaluation”, Bonn, 19-20 March 2007, available at: 
www.oecd.org/dataoecd/43/60/38405938.pps. 



Evaluation of communication: the case of the 2004 Asian Tsunami 

The 2004 Asian Tsunami put official donors and NGOs into the public 
spotlight as never before, not least because of the timing (day after Christmas) 
and the extraordinary reach and destruction of the catastrophe. Given the 
unprecedented public attention, donors were acutely aware that failure to deliver 
on public expectations could damage the credibility of the “aid community” and 
dramatically increase scepticism about development co-operation in general. 
They knew that their performance in dealing with the disaster would be judged 
upon their promptness to: i) express compassion; ii) send representatives in 
the region; and Hi) offer aid. The sense of urgency also required donors to make 
special efforts to communicate about the differences between i) emergency relief; 
ii) bilateral humanitarian aid delivered through NGOs; Hi) longer-term assistance 
to reconstruction. The media contributed to this demand from public opinion as 
the Tsunami and issues about aid effectiveness stayed in the headlines over the 
long term with only a slow gradual disappearance from the news. 
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Good Practice 2: AusAID’s Communication about the Tsunami 

(2005 and 2006) 

AusAID commissioned an external evaluator to undertake an ex-post evaluation of how 
its Public Affairs Group (PAG) reacted to the Tsunami disaster with a view to integrating 
lessons learnt into future “preparedness” for these kinds of disasters. 

The purpose of the evaluation was to examine how the PAG dealt with the communication 
challenge and not the effectiveness of the communication itself. The main tools used 
were a survey and a facilitated discussion with staff including management. 

An open-ended survey was emailed before the facilitation to enable members of PAG 
to confidentially express opinions on: 

• what went right? 

• what went wrong? 

• what needed improving? 

The twelve specific recommendations that were made for future strategies evolved 
from the survey and discussion. They include: having a crisis management plan for public 
affairs; a staffing plan; overseas deployment arrangements; the public affairs groups 
should actively participate in the Agency’s strategic meetings when managing such crises; 
consider better branding in the field; develop a media strategy and benchmarking with 
other agencies. 

The overriding best practices emerging from the evaluation for AusAID are recognition 
that good communication is as important as the provision of aid during emergencies 
like the Tsunami crisis; management should present their expectations on how Public 
Affairs should communicate in the event of a disaster; and technology capability and 
support in the field are very important. 

Source: AusAID (2005), Public Affairs Group response to Tsunami, by Clarity Communications Australia Pty Ltd, 
unpublished. 



Evaluation of advocacy and campaigning: the case of the Moke Poverty History 
campaign 

The Make Poverty History campaign (2005) was initiated by a coalition of UK 
NGOs to coincide with the UK Presidencies of the EU and G8, the UN Summit 
five years after the Millennium Goals and WTO negotiations. Inspired by lessons 
learnt from the Jubilee 2000 Campaign, Make Poverty History was composed of a 
broad coalition of British NGOs which grew to 540 members representing most 
sections of UK civil society. The objectives of the campaign included: achieving 
policy change in 2005 in the areas of aid, debt relief and trade justice; creating 
unstoppable momentum for change in 2005; and leaving the public committed to 
further change beyond 2005. Public mobilisation was considered to be the greatest 
achievement of the campaign. 
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Good Practice 3: The Make Poverty History Campaign, United Kingdom 

The Make Poverty History 2005 Campaign Evaluation was commissioned to answer 
three questions: 

1) What progress did the coalition make against its objectives during 2005? 

2) What were the strengths and weaknesses of the coalition’s approach and set-up? 

3) What lessons can be learned for the future? 

The method and tools for the evaluation were to interview over 70 different stakeholders, 
review various documentation made available by the staff, questionnaires, incorporating 
other evaluation work under way and quantitative and qualitative work done on public 
perceptions of poverty. 

The evaluation gives a lot of emphasis to process-related matters of the campaign. 

Limitations of the evaluation: in their report the evaluators emphasised the difficulty of 
getting a representative sample due to time and budget constraints, and the scale and 
diversity of such a large-scale campaign. It was also felt that more detailed evaluation of 
public awareness was necessary to access the long-term impact, if any, of the campaign in 
changing attitudes to poverty. Even if the long-term impact of the campaign was neither 
its objective nor that of the evaluation, there is always pressure to demonstrate an 
impact in terms of what the public learnt. But very often this is not the main part of an 
evaluation and the dilemma of demonstrating impact on public opinion and knowledge 
is not solved. 

Source: Martin et o/., 2005: www.firetail.co.ul</MPH_2005_Evaluation.pdf.. 



Evaluation of development and global education: the case of NGO work in 

German schools 

Evaluations of development and global education deal with the difficulty of 
identifying the impact of the project on learning by focusing on concrete results 
— as the number of tool kits printed, lessons held in school, the number of students 
reached, and the self-perception of the people involved — instead of the learning 
outcome in terms of changed behaviour or conceptions. 

There are many ways to deliver development education through the educational 
system, either formally in schools at all levels, less formally through adult education, 
further vocational training, or in informal learning situations such as fairtrade shops. 
Evaluation in development education may focus on the activities themselves or 
on the system (Hock, 2004). The European Peer Review Process by GENE is an 
important example of a system level evaluation in this field^^. 
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Good Practice 4: Education about Children’s Rights, 

“Terre des hommes” Germany 

From June 2004 to May 2006 the German NGO “Terre des hommes’’ trained about 
90 university students in development education, taking examples from its own work 
and focusing on children’s rights. These students then offered lessons in school to 
sensitise 9 to 15-year-old students in public schools and motivate them to get involved 
in development co-operation. About 2 100 students in classes were reached. The NGO 
asked for an ex-post evaluation focusing on the impact of this facilitator programme 
concerning: 

1) the facilitators; 

2) the teachers involved; 

3) the participating classes. 

The main tools used were questionnaires and group discussions. The questionnaires 
focused on the facilitators’ training and their work in schools, e.g. the intensity of 
preparation and post-processing of their school visits and the results of the activities in 
class.The teachers were asked to rate their satisfaction with the facilitators’ engagement 
and their long term impact. The group discussions, conducted with the facilitators, 
showed their perceptions and concepts of development co-operation as well as their 
identification with the work of the NGO and their working conditions. 

The evaluation concluded that there had been an important impact by training facilitators. 
The effectiveness and impact of the NGO in development education improved significantly. 
The learning process made an important difference to the facilitator themselves. However, 
the evaluation also showed that activities like these, even when reaching many students 
in two or three hours of lessons and activities in schools, do not lead to measurable 
learning outcomes.The evaluation thus helped to balance expectations. 

The multi-level approach of this evaluation is particularly interesting. It linked three 
different “customers’’: the trained facilitators, the teacher, and the students themselves. 
This multi-level approach resulted in greater understanding about the different factors 
that contribute to quality development education. For example, the necessity of training 
the facilitators not only about the work of the NGO and children's rights but also about 
the learning needs and the competences of the students. 

Source: Bergmuller, C and S. Hock (2006), Evaluationsbericht des Multiplikatorenprogramm Nordrhein-Westfalen von Terre 
des hommes, Ntirnberg 



Opinion polling: using it to measure aggregate performance 

A lesson learnt from evaluations is their limited potential for assessing the 
impact of an activity on public awareness and learning. Public opinion polls may 
provide indicators about awareness, which can be tested for correlations with 
awareness raising and learning strategies (Fransman et o/., 2004). However, they 
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are not always the magic solution to measuring impact because surveys are rarely 
conducted both before and after the activity due to expense, time and, sometimes, 
poor planning. In addition, surveys are often embroiled in a political agenda looking 
for endorsement of policies. They do not always contribute to, or serve as an 
evaluation tool for, a communication strategy. Furthermore, cost is often an 
obstacle to conducting comprehensive and longitudinal opinion surveys. 



Good Practice 5: Public Support Pentagram from NCDO 

The objective of the communication strategy of the Dutch National Committee for 
International Cooperation and Sustainable Development (NCDO) is to strengthen 
public support for international co-operation, defined as “the total of the commitment 
to, and the support within the Netherlands for, the goals of international co-operation’’. 
In order to measure this, monitoring and evaluation are integrated in the communication 
strategy. The public support pentagram is the tool developed by NCDO for measuring 
performance over time. 





2006 

2007 

2008 

2009 

2010 
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Good Practice 5 (contd.) 

The indicators of growing support are people’s knowledge, attitudes and behaviour 
with respect to the goals of international co-operation. The indicators for behaviour 
are participation in and the development of civil society in the Netherlands. 

These indicators are measured under five categories: politics, knowledge, opinion, 
participation and development civil society. Each category has three indicators. For the 
participation category the indicators are: 

1) The number of people involved in small-scale private activities. 

2) The number of television programmes in the field of international co-operation; 

3) The market share of products from the fair trade sector. 

Surveys conducted in 2006 served as the baseline measurement. From this baseline 
measurement realistic projections were made for the years 2007 to 20 10. These 
expectations will be taken as the reference for public opinion polls in 2007 to 2010 to 
evaluate the impact of the communication strategy in the following years. 

Source: Bergmans et o/., 2007, available at: www.ncdo.nl 



Conclusion 

The international agenda to inform and educate citizens in OECD countries 
about global poverty and development has reached unprecedented heights since 
the beginning of the millennium. This brings new opportunities and challenges to 
the communicators and educators who have been promoting this work for decades. 
Evaluation is at the heart of these challenges. 

Solutions and proposals on how to approach evaluation challenges were 
discussed at an Experts’ Workshop on Evaluation by evaluators, donors and NGO 
practitioners. Process-related challenges on the nuts and bolts of how to do an 
evaluation are much easier to overcome than the policy challenges. They are 
well documented and tools such as the evaluation cycle help keep the evaluator 
on track and systematic. Consultation and transparency between the evaluator, 
practitioners and the funder — when it is external — throughout the evaluation 
are keys to a smooth process. Principles of transparency are also crucial for the 
“heavyweight” policy challenges such as working in partnership. Designing evaluations 
that fulfil both accountability and lesson learning requirements are fundamental 
to establishing a positive culture of evaluation on public awareness and learning 
about development. 
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This Policy Brief has touched upon many of the stumbling-blocks that help 
explain the reticence and inertia that can be found when it comes to mainstreaming 
evaluation in public awareness raising and learning about global developmentThe 
case study examples demonstrate both the methodologies for and the benefits 
from evaluating information, advocacy and development education activities. Still, 
the Policy Brief only begins to scratch the surface in terms of identifying what 
constitutes best practice in evaluating communication, advocacy and development 
education. 

In taking their work on evaluation forward, an intermediary next step for 
DAC members participating in the Informal Network of DAC Development 
Communicators along with their partners could include: creating a space (e.g. website) 
to share perspectives, experiences and results from evaluation; pooling resources 
for scientific research on the long-term impact of public awareness-raising activities; 
building a stronger knowledge base on what works and what doesn’t work in this 
sector; and finally, working together to develop minimum standards for evaluation 
of communication, advocacy and education about global development. 
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DEA, London. See Introduction. 

1 8. Stufflebeam and Shinkfield, 2007, pp. 326-346. 

1 9. In advocacy and campaigning there is a debate about what indicators could best describe the 
quality of a campaign. Should it be the political reaction? Or newspaper coverage?The number 
of participants in demonstrations? The number of members of parliament signing the motion 
of the campaign? In many cases all of these indicators are used. 

20. ITECO presentation at Experts’ Workshop on “Development Communication, Advocacy and 
Education: Tips and Tools for Improving Evaluation’’, Bonn, 19-20 March 2007, www.oecd. 
org/dataoecd/43/57/38405984.pps#6_. 

2 1 . CCIC consultation on evaluation, ITECO on tensions, DEEEP reports www.deeep.org/english/ 
about_deeep/training/index.php. 

22. See North-South Centre of the Council of Europe, 2005 ff; O’Loughlin, 2004. 
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accessible way. This series, with its wide, targeted and rapid 
distribution, is specifically intended for policy and decision makers 
in the fields concerned. 

This Brief introduces OECD-based policy makers and practitioners 
— active in the fields of communication, advocacy and education 
about poverty, inequality and development co-operation — to the 
need for more evaluation of their work. It militates for creating a 
culture of learning for all public awareness-raising work; it presents 
the basic elements and key principles for creating this culture; 
and addresses complex questions that cause inertia in putting 
evaluation into practice. Questions include: Can an evaluation 
measure the impact of a campaign on public attitudes? Who 
should pay for the evaluation and how much should it cost? How 
to approach resistance to and fear of evaluation? Finally, the Brief 
proposes strengthened international collaboration to promote 
the evaluation of actions to raise public awareness about poverty, 
inequality and development co-operation; to pool resources for 
these evaluations; share best practice and lessons; and develop 
minimum standards for evaluation in this sector.. 
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